Orthogonal neural codes for speech in the infant brain.

Creating invariant representations from an everchanging speech signal is a major challenge for the human brain. Such an ability is particularly crucial for preverbal infants who must discover the phonological, lexical, and syntactic regularities of an extremely inconsistent signal in order to acquire language. Within the visual domain, an efficient neural solution to overcome variability consists in factorizing the input into a reduced set of orthogonal components. Here, we asked whether a similar decomposition strategy is used in early speech perception. Using a 256-channel electroencephalographic system, we recorded the neural responses of 3-mo-old infants to 120 natural consonant-vowel syllables with varying acoustic and phonetic profiles. Using multivariate pattern analyses, we show that syllables are factorized into distinct and orthogonal neural codes for consonants and vowels. Concerning consonants, we further demonstrate the existence of two stages of processing. A first phase is characterized by orthogonal and context-invariant neural codes for the dimensions of manner and place of articulation. Within the second stage, manner and place codes are integrated to recover the identity of the phoneme. We conclude that, despite the paucity of articulatory motor plans and speech production skills, pre-babbling infants are already equipped with a structured combinatorial code for speech analysis, which might account for the rapid pace of language acquisition during the first year.

Mental compression of spatial sequences in human working memory using numerical and geometrical primitives.

How does the human brain store sequences of spatial locations? We propose that each sequence is internally compressed using an abstract, language-like code that captures its numerical and geometrical regularities. We exposed participants to spatial sequences of fixed length but variable regularity while their brain activity was recorded using magneto-encephalography. Using multivariate decoders, each successive location could be decoded from brain signals, and upcoming locations were anticipated prior to their actual onset. Crucially, sequences with lower complexity, defined as the minimal description length provided by the formal language, led to lower error rates and to increased anticipations. Furthermore, neural codes specific to the numerical and geometrical primitives of the postulated language could be detected, both in isolation and within the sequences. These results suggest that the human brain detects sequence regularities at multiple nested levels and uses them to compress long sequences in working memory.

Probing the limits of activity-silent non-conscious working memory.

Two types of working memory (WM) have recently been proposed: (i) active WM, relying on sustained neural firing, and (ii) activity-silent WM, for which firing returns to baseline, yet memories may be retained by short-term synaptic changes. Activity-silent WM in particular might also underlie the recently discovered phenomenon of non-conscious WM, which permits even subliminal stimuli to be stored for several seconds. However, whether both states support identical forms of information processing is unknown. Theory predicts that activity-silent states are confined to passive storage and cannot operate on stored information. To determine whether an explicit reactivation is required before the manipulation of information in WM, we evaluated whether participants could mentally rotate brief visual stimuli of variable subjective visibility. Behaviorally, even for unseen targets, subjects reported the rotated location above chance after several seconds. As predicted, however, at the time of mental rotation, such blindsight performance was accompanied by (i) neural signatures of consciousness in the form of a sustained desynchronization in alpha/beta frequency and (ii) a reactivation of the memorized information as indicated by decodable representations of participants' guess and response. Our findings challenge the concept of genuine non-conscious "working" memory, argue that activity-silent states merely support passive short-term memory, and provide a cautionary note for purely behavioral studies of non-conscious information processing.

Temporal-order information can be maintained in non-conscious working memory.

Classical theories hold conscious perception and working memory to be tightly interwoven. Recent work has challenged this assumption, demonstrating that information may be stored for several seconds without any subjective awareness. Does such non-conscious working memory possess the same functional properties as conscious working memory? Here, we probe whether non-conscious working memory can maintain multiple items and their temporal order. In a visual masking task with a delayed response, 38 participants were asked to retain the location and order of presentation of two sequentially flashed spatial positions, and retrieve both after a 2.5 second delay. Even when subjective visibility was nil, subjects' objective forced-choice performance exceeded chance level and, crucially, distinct retrieval of the first and second location was observed on both conscious and non-conscious trials. Non-conscious working memory may therefore store two items in proper temporal order. These findings can be explained by recent models of activity-silent working memory.

Conscious Perception: Time for an Update?

Understanding the neural mechanisms underlying conscious perception has become a central endeavor in cognitive neuroscience. In theories of conscious perception, a stimulus gaining conscious access is usually considered as a discrete neuronal event to be characterized in time or space, sometimes referred to as a conscious "episode." Surprisingly, the alternative hypothesis according to which conscious perception is a dynamic process has rarely been considered. Here, we discuss this hypothesis and its implications. We show how it can reconcile inconsistent empirical findings on the timing of the neural correlates of consciousness and make testable predictions. According to this hypothesis, a stimulus is consciously perceived for as long as it is recoded to fit an ongoing stream composed of all other perceived stimuli. We suggest that this "updating" process is governed by at least three factors (1) context, (2) stimulus saliency, and (3) observers' goals. Finally, this framework forces us to reconsider the typical distinction between conscious and unconscious information processing.

Discrete and continuous mechanisms of temporal selection in rapid visual streams.

Humans can reliably detect a target picture even when tens of images are flashed every second. Here we use magnetoencephalography to dissect the neural mechanisms underlying the dynamics of temporal selection during a rapid serial visual presentation task. Multivariate decoding algorithms allow us to track the overlapping brain responses induced by each image in a rapid visual stream. The results show that temporal selection involves a sequence of gradual followed by all-or-none stages: (i) all images first undergo the same parallel processing pipeline; (ii) starting around 150 ms, responses to multiple images surrounding the target are continuously amplified in ventral visual areas; (iii) only the images that are subsequently reported elicit late all-or-none activations in visual and parietal areas around 350 ms. Thus, multiple images can cohabit in the brain and undergo efficient parallel processing, but temporal selection also isolates a single one for amplification and report.

A theory of working memory without consciousness or sustained activity.

Working memory and conscious perception are thought to share similar brain mechanisms, yet recent reports of non-conscious working memory challenge this view. Combining visual masking with magnetoencephalography, we investigate the reality of non-conscious working memory and dissect its neural mechanisms. In a spatial delayed-response task, participants reported the location of a subjectively unseen target above chance-level after several seconds. Conscious perception and conscious working memory were characterized by similar signatures: a sustained desynchronization in the alpha/beta band over frontal cortex, and a decodable representation of target location in posterior sensors. During non-conscious working memory, such activity vanished. Our findings contradict models that identify working memory with sustained neural firing, but are compatible with recent proposals of 'activity-silent' working memory. We present a theoretical framework and simulations showing how slowly decaying synaptic changes allow cell assemblies to go dormant during the delay, yet be retrieved above chance-level after several seconds.

Time-Resolved Decoding of Two Processing Chains during Dual-Task Interference.

The human brain exhibits fundamental limitations in multitasking. When subjects engage in a primary task, their ability to respond to a second stimulus is degraded. Two competing models of multitasking have been proposed: either cognitive resources are shared between tasks, or they are allocated to each task serially. Using a novel combination of magneto-encephalography and multivariate pattern analyses, we obtained a precise spatio-temporal decomposition of the brain processes at work during multitasking. We discovered that each task relies on a sequence of brain processes. These sequences can operate in parallel for several hundred milliseconds but beyond ∼ 500 ms, they repel each other: processes evoked by the first task are shortened, while processes of the second task are either lengthened or postponed. These results contradict the resource-sharing model and further demonstrate that the serial model is incomplete. We therefore propose a new theoretical framework for the computational architecture underlying multitasking.

Subjective report of eye fixations during serial search.

Humans readily introspect upon their thoughts and their behavior, but how reliable are these subjective reports? In the present study, we explored the consistencies of and differences between the observer's subjective report and actual behavior within a single trial. On each trial of a serial search task, we recorded eye movements and the participants' beliefs of where their eyes moved. The comparison of reported versus real eye movements revealed that subjects successfully reported a subset of their eye movements. Limits in subjective reports stemmed from both the number and the type of eye movements. Furthermore, subjects sometimes reported eye movements they actually never made. A detailed examination of these reports suggests that they could reflect covert shifts of attention during overt serial search. Our data provide quantitative and qualitative measures of observers' subjective reports and reveal experimental effects of visual search that would otherwise be inaccessible.

How does the extraction of local and global auditory regularities vary with context?

How does the human brain extract regularities from its environment? There is evidence that short range or 'local' regularities (within seconds) are automatically detected by the brain while long range or 'global' regularities (over tens of seconds or more) require conscious awareness. In the present experiment, we asked whether participants' attention was needed to acquire such auditory regularities, to detect their violation or both. We designed a paradigm in which participants listened to predictable sounds. Subjects could be distracted by a visual task at two moments: when they were first exposed to a regularity or when they detected violations of this regularity. MEG recordings revealed that early brain responses (100-130 ms) to violations of short range regularities were unaffected by visual distraction and driven essentially by local transitional probabilities. Based on global workspace theory and prior results, we expected that visual distraction would eliminate the long range global effect, but unexpectedly, we found the contrary, i.e. late brain responses (300-600 ms) to violations of long range regularities on audio-visual trials but not on auditory only trials. Further analyses showed that, in fact, visual distraction was incomplete and that auditory and visual stimuli interfered in both directions. Our results show that conscious, attentive subjects can learn the long range dependencies present in auditory stimuli even while performing a visual task on synchronous visual stimuli. Furthermore, they acquire a complex regularity and end up making different predictions for the very same stimulus depending on the context (i.e. absence or presence of visual stimuli). These results suggest that while short-range regularity detection is driven by local transitional probabilities between stimuli, the human brain detects and stores long-range regularities in a highly flexible, context dependent manner.

Toward a computational theory of conscious processing.

The study of the mechanisms of conscious processing has become a productive area of cognitive neuroscience. Here we review some of the recent behavioral and neuroscience data, with the specific goal of constraining present and future theories of the computations underlying conscious processing. Experimental findings imply that most of the brain's computations can be performed in a non-conscious mode, but that conscious perception is characterized by an amplification, global propagation and integration of brain signals. A comparison of these data with major theoretical proposals suggests that firstly, conscious access must be carefully distinguished from selective attention; secondly, conscious perception may be likened to a non-linear decision that 'ignites' a network of distributed areas; thirdly, information which is selected for conscious perception gains access to additional computations, including temporary maintenance, global sharing, and flexible routing; and finally, measures of the complexity, long-distance correlation and integration of brain signals provide reliable indices of conscious processing, clinically relevant to patients recovering from coma.

Reducing multi-sensor data to a single time course that reveals experimental effects.

BACKGROUND: Multi-sensor technologies such as EEG, MEG, and ECoG result in high-dimensional data sets. Given the high temporal resolution of such techniques, scientific questions very often focus on the time-course of an experimental effect. In many studies, researchers focus on a single sensor or the average over a subset of sensors covering a "region of interest" (ROI). However, single-sensor or ROI analyses ignore the fact that the spatial focus of activity is constantly changing, and fail to make full use of the information distributed over the sensor array. METHODS: We describe a technique that exploits the optimality and simplicity of matched spatial filters in order to reduce experimental effects in multivariate time series data to a single time course. Each (multi-sensor) time sample of each trial is replaced with its projection onto a spatial filter that is matched to an observed experimental effect, estimated from the remaining trials (Effect-Matched Spatial filtering, or EMS filtering). The resulting set of time courses (one per trial) can be used to reveal the temporal evolution of an experimental effect, which distinguishes this approach from techniques that reveal the temporal evolution of an anatomical source or region of interest. RESULTS: We illustrate the technique with data from a dual-task experiment and use it to track the temporal evolution of brain activity during the psychological refractory period. We demonstrate its effectiveness in separating the means of two experimental conditions, and in significantly improving the signal-to-noise ratio at the single-trial level. It is fast to compute and results in readily-interpretable time courses and topographies. The technique can be applied to any data-analysis question that can be posed independently at each sensor, and we provide one example, using linear regression, that highlights the versatility of the technique. CONCLUSION: The approach described here combines established techniques in a way that strikes a balance between power, simplicity, speed of processing, and interpretability. We have used it to provide a direct view of parallel and serial processes in the human brain that previously could only be measured indirectly. An implementation of the technique in MatLab is freely available via the internet.

Distinct brain mechanisms for conscious versus subliminal error detection.

Metacognition, the ability to monitor one's own cognitive processes, is frequently assumed to be univocally associated with conscious processing. However, some monitoring processes, such as those associated with the evaluation of one's own performance, may conceivably be sufficiently automatized to be deployed non-consciously. Here, we used simultaneous electro- and magneto-encephalography (EEG/MEG) to investigate how error detection is modulated by perceptual awareness of a masked target digit. The Error-Related Negativity (ERN), an EEG component occurring ~100 ms after an erroneous response, was exclusively observed on conscious trials: regardless of masking strength, the amplitude of the ERN showed a step-like increase when the stimulus became visible. Nevertheless, even in the absence of an ERN, participants still managed to detect their errors at above-chance levels under subliminal conditions. Error detection on conscious trials originated from the posterior cingulate cortex, while a small response to non-conscious errors was seen in dorsal anterior cingulate. We propose the existence of two distinct brain mechanisms for metacognitive judgements: a conscious all-or-none process of single-trial response evaluation, and a non-conscious statistical assessment of confidence.

A shared cortical bottleneck underlying Attentional Blink and Psychological Refractory Period.

Doing two things at once is difficult. When two tasks have to be performed within a short interval, the second is sharply delayed, an effect called the Psychological Refractory Period (PRP). Similarly, when two successive visual targets are briefly flashed, people may fail to detect the second target (Attentional Blink or AB). Although AB and PRP are typically studied in very different paradigms, a recent detailed neuromimetic model suggests that both might arise from the same serial stage during which stimuli gain access to consciousness and, as a result, can be arbitrarily routed to any other appropriate processor. Here, in agreement with this model, we demonstrate that AB and PRP can be obtained on alternate trials of the same cross-modal paradigm and result from limitations in the same brain mechanisms. We asked participants to respond as fast as possible to an auditory target T1 and then to a visual target T2 embedded in a series of distractors, while brain activity was recorded with magneto-encephalography (MEG). For identical stimuli, we observed a mixture of blinked trials, where T2 was entirely missed, and PRP trials, where T2 processing was delayed. MEG recordings showed that PRP and blinked trials underwent identical sensory processing in visual occipito-temporal cortices, even including the non-conscious separation of targets from distractors. However, late activations in frontal cortex (>350 ms), strongly influenced by the speed of task-1 execution, were delayed in PRP trials and absent in blinked trials. Our findings suggest that PRP and AB arise from similar cortical stages, can occur with the same exact stimuli, and are merely distinguished by trial-by-trial fluctuations in task processing.

Mapping introspection's blind spot: reconstruction of dual-task phenomenology using quantified introspection.

Psychologists often dismiss introspection as an inappropriate measure, yet subjects readily volunteer detailed descriptions of the time and effort that they spent on a task. Are such reports really so inaccurate? We asked subjects to perform a psychological refractory period experiment followed by extensive quantified introspection. On each trial, just after their objective responses, subjects provided no less than four subjective estimates of the timing of sensory, decision and response events. Based on these subjective variables, we reconstructed the phenomenology of an average trial and compared it to objective times and to predictions derived from the central interference model. Introspections of decision time were highly correlated with objective measures, but there was one point of drastic distortion: subjects were largely unaware that the second target was waiting while the first task was being completed, the psychological refractory period effect. Thus, conscious perception is systematically delayed and distorted while central processing resources are monopolized by another task.

New object onsets reduce conscious access to unattended targets.

Attention to a visual target can affect perception of a subsequent target for half a second, increasing its sensitivity to backward masking (the attentional blink, AB). In 6 studies, we compared the AB when the second target and its mask had a common onset and when the mask appeared after the target. The results indicate that common-onset masks do not produce large ABs even when there is a feature change or an interruption of the mask after the target but do produce a large AB if the location of the mask is changed. The data suggest that new object onsets reduce conscious access to unattended targets.